
LEARNING POINT 



North Central Regional 
Educational Laboratory 



No. 16 June 2004 



Research-Based Analysis of 
Education Policy 



POLICY ISSUES 



STATE EDUCATION DATA SYSTEMS THAT INCREASE 
LEARNING AND IMPROVE ACCOUNTABILITY 

By Robert M. Palaich, Ph.D., Dixie Griffin Good, and A rie van der Ploeg 



Every state in the nation is taking a long, hard look at education data and how it can be collected, housed, analyzed, accessed, 
and used better than ever before. 

Since the advent of the accountability movement in education reform, state education agencies (SEAs) have encountered signif- 
icant data-collection and data-management issues. Most states have tried to meet these challenges head on, but, more often 



than not, demands have outstripped capacity as data needs con- 
tinue to evolve and grow more complex. Policymakers will want 
to protect and improve their state's information infrastructure in 
order to realize the full benefits of standards-based reform. 

Future data management will require not only making the best 
use of current data but also tailoring student-performance data 
for district, school, teacher, parent, and student use. Further 
investments in the technology and human infrastructure under- 
pinning education accountability and improvement efforts will 
be necessary. However, most elected officials will not have the 
expertise to grasp the technical and operational trade-offs asso- 
ciated with these investments, and it is not reasonable to expect 
information technology professionals to mobilize the political 
coalitions necessary to support these critical investments. 

Nevertheless, states need thoughtful mechanisms for meeting 
evolving education data needs. As state and federal accounta- 
bility and reporting requirements grow in scope — and as 
decisions about how, when, and where to allocate new educa- 
tion resources are increasingly being determined on the basis of 
the best available data — states can no longer afford to wait with 
building high-quality data systems. In addition, the integration of 
various social services into schools, the extension of the school 
day, and the lengthening of the K-12 education system to include 
prekindergarten and postsecondary transitions require state edu- 
cation data systems to have the flexibility to work with other 



ISSUE OVERVIEW 

Driven by growing accountability pressures, states 
and districts have invested in a variety of computer- 
ized systems for data storage, analysis, and reporting. 
As accountability policies demand access to more 
transparent and accurate data about every aspect of 
the education process, developing linkages among his- 
torically disparate systems is becoming more critical. 

This edition of Policy Issues reviews the current con- 
dition of state data systems by looking critically at the 
past, present, and future of education data use to help 
build an understanding of an ideal data system. This 
paper examines the components needed to address 
system improuements and provides policy recommen- 
dations to help states create efficient and useful data 
systems that commit to advancing accountability sys- 
tems to improve student learning. 

— Jill Shively, Learning Point Associates 
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Address the key tasks associated with effective use 
of education data. (See p. 9 for complete list of five tasks.) 

Create an education data oversight commission 

in each state. 

Adopt common data standards, data- collection 
guidelines, and data-management practices. 



POLICY RECOMMENDATIONS 



comparable state databases. This paper reviews past, 
present, and future uses of education data and 
addresses why state education leaders should take 
action to build and maintain optimal education data 
systems. Finally, recommendations and specific strate- 
gies for getting there are presented. 

EDUCATION DATA: WHO NEEDS WHAT 

First, a look at the consumers of education data and 
the types of data they need is in order. Each level of 
governance, implementation, and participation has 
its own uses for student results data. From the U.S. 
Department of Education to the classroom teacher 
in each school building, the right data at the right 
time can play an important role in effective decision 
making. Table 1 shows some of the education data 
needs of various audiences, and how such data can 
be used to improve education results. 

The No Child Left Behind (NCLB) Act of 2001 
required states to disaggregate test-score data in 
order to track the progress of all students and stu- 
dent groups (NCLB, 2002). Many states were 
already doing this when the NCLB legislation was 
passed. Furthermore, this requirement was not new: 



The 1994 Elementary and Secondary Education Act 
(ESEA) had requested this type of accountability 
information. The difference lies in the consequences 
that the NCLB Act attaches to the results of this 
data. Schools now need to study and interpret the 
data, and they need to ensure that underperforming 
groups make progress. 

The Maryland school improvement Web site 
(www.mdk12.org) broke new ground when it went 
public in May 2000. Begun as an Office of Educational 
Research and Improvement (OERI) funded grant to 
the Maryland State Department of Education in the 
late 1990s, mdk12.org was the first large-scale instan- 
tiation of a Web-based, data-driven process to school 
improvement. From the beginning, it has featured 
interactive graphical data disaggregation, down to 
the building level. Longitudinal patterns are an impor- 
tant aspect of the data reporting. Particularly 
significant is the fact that the data tools are always 
paired with a focus on state standards and resources 
for improving instructional practice. 

In 1999, the North Central Regional Educational 
Laboratory (NCREL) negotiated with the new state 
superintendent in Illinois to build a similar capacity 









U.S. Department of 
Education 


• Disaggregated achievement results 
by subgroup 

• Adequate yearly progress (AYP) for 
each school and district 

• Teacher qualifications 

• School report cards 

• Program expenditures 

• Program enrollments 


• Ensure No Child Left Behind 
(NCLB) compliance 

• Conduct research and inform 
improvement efforts 











State Policymakers and 
Education Agencies 


• Standardized state test scores 

• Percentages of students achieving 
proficiency 

• School report cards 


• Confirm state standards are being met 

• Recognize achievement 

• Provide rewards or sanctions 

• Provide technical assistance to 
districts and schools 

• Develop program design 

• Inform school choice 


District Leaders 


• Percentages of students achieving 
proficiency — by building, by 
subgroup 

• Aggregated longitudinal student- 
achievement data 

• School report cards 


• Help parents and community focus 
on student achievement 

• Provide technical assistance to 
schools 

• Prepare for NCLB and AYP 
consequences 

• Inform school choice 

• Make districtwide curriculum 
decisions 


School Leaders 


• Percentages of students achieving 
proficiency — by grade, by program, 
by teacher, by subgroup 

• School report cards 

• Disaggregated longitudinal 
student-achievement records 

• Attendance data 

• Graduation rates 

• Individual student-performance 
records 


• Help school community focus on 
student achievement 

• Focus staff use of time 

• Inform school choice 

• Flag students in need of assistance 

• Make schoolwide curriculum decisions 


Classroom Teachers 


• Percentages of student subgroups 
achieving proficiency 

• Individual student subtest scores 

• School report cards 

• Individual longitudinal student- 
achievement trends 

• Attendance 

• Student performance in prior 
and subsequent grades 

• Diagnostic information on 
students' learning needs 


• Help students focus on achievement 

• Focus staff use of time 

• Flag students in need of assistance 

• Make classroom curriculum decisions 

• Create additional assessment items 


Students and Parents 


• Grades on assignments and tests 

• Portfolio work 

• Individual longitudinal achievement 
record 

• Diagnostic information on students' 
learning needs 

• School report cards 


• Help students focus on achievement 

• Inform progress against proficiency 
standards 

• Inform school choice 


Community and 
Business and Industry 


• Percentages of students and 
subgroups achieving proficiency 

• School report cards 


• Help parents and community to 
focus on student achievement 

• Provide assistance to needy schools 



This table specifically targets student data, but it is understood that student data is not the only important education data. 



POLICY ISSUES No. 16, June 2004 




POLICY ISSUES No. 16, June 2004 



for that state. The Illinois School Improvement Web 
site (ilsi.isbe.net) went live in November 2001. 
NCREL also collaborated with the Department of 
Public Instruction in Wisconsin, which led to 
Wisconsin's Information Network for Successful 
Schools (WINSS) Web site (www.dpi.state.wi. us/sig) 
going public in December 2001 . Minnesota's School 
& District Information/Analysis (education. state. 
mn.us/html/intro_mde_analysis.htm) followed in 
January 2002; it too was a joint effort with NCREL. 
By the summer of 2002, the Indiana Accountability 
System for Academic Progress (ASAP) Web site was 
accessible (www.doe.state.in.us/asap/welcome.html). 
In 2003, ASAP received strong praise from 
President George W. Bush and Secretary of 
Education Rod Paige. 

Education data can inform policy and practice deci- 
sions and lead to continuous improvement of the 
kind espoused by Baldrige (Illinois Business 
Roundtable, 2002), Bernhardt (1998; 2004), Fullan 
(1991), Senge (Senge, Kleiner, Roberts, Ross, & 
Smith, 1994), and others. Educators and policymak- 
ers need systematic analysis of the data collected on 
the effectiveness of various interventions and pilot 
programs designed to improve student perform- 
ance. Although education leaders should be able to 
search for scientifically based research from their 
state as well as from other states, this search often 
occurs on a broader level and can be a very time- 
consuming and arduous process. While national 
research programs such as the What Works 
Clearinghouse (www.w-w-c.org) will soon provide 
policymakers and practitioners with a growing 
research base from which to build effective reforms 
(and comply with NCLB regulations for federal fund- 
ing), only close inspection of state and district data 
will find the patterns and relationships that support 
and enable effective local decisions and consistent 
data-driven growth. 

Data-informed decision making is only beginning to 
emerge in education practice. States can certainly use 
targeted data collection and analysis to inform educa- 
tion policy and practice. Furthermore, states must play 
a role in making this data analysis available and useful 
to schools and districts that lack the resources to do it 
themselves. This latter role has increased sharply in 
emphasis and is one for which information technology 
professionals are poorly prepared. 



UPGRADING THE SYSTEM: PAST, 
PRESENT, AND FUTURE USES OF 
EDUCATION DATA 

Every time a policymaker is confronted with the 
prospect of another technology expenditure, no 
doubt he or she is tempted to ask, "Again? Didn't 
we just deal with this a few years back? When will we 
get this fixed once and for all?" And, no doubt, the 
technical data managers are tempted to reply, "Yes, 
again. Yes, we did," and "We won't — costs for data 
system maintenance and upgrades are ongoing." 
The questions are valid, and the answers are accu- 
rate, if oversimplified. 

The factors driving today's need for investing anew 
in education data systems are many, ranging from 
continuing the contribution of legacy systems to 
emerging data consumers to new kinds of perform- 
ance measures and data standards. A closer look at 
some of the forces driving change in this area follows. 

The Roots of 
Education Data Systems 

First, it is useful to review how state data systems 
came into being. Most early data systems were cre- 
ated to manage SEA financial and budgeting 
information and personnel records. Tracking district 
compliance came next, followed only relatively 
recently by tracking student-assessment data. These 
discrete databases often resided in hardware or 
applications that were proprietary and incompatible 
with others. As a result, vast stores of data may either 
be inaccessible for new or different uses or of only 
limited value, resulting in gross inefficiencies: redun- 
dant data entry, higher support costs, and inability to 
access pertinent data for decision making. 

Past well-intentioned decisions frequently limit 
future options. For example, confidentiality of stu- 
dent records is a major concern of SEAs, and 
appropriately so. To ensure that SEAs themselves 
could not be responsible for inadvertent release of 
individual records, some states, by law or by proce- 
dure, refused to store such records in their own 
databases. Test publishers held the source files 
(e.g., until very recently in Wisconsin), or intermedi- 
ate units were created to store and process 
student-performance data files (e.g., in Ohio). As a 
consequence, building data structures that link at 
the student level — to provide the detailed analyses 
now expected — has proven exceedingly difficult. 




Even as new demands and various uses for data 
arise, legacy systems command the attention of 
data managers. They deal with multiple platforms, 
multiple programming languages, hardware limita- 
tions, and idiosyncratic codes and tags. 

States look to vendors for turnkey solutions. 
Proprietary solutions, though relatively quick to imple- 
ment, may lead to new sets of problems in sharing 
data. Hardware and applications vary considerably 
from state to state: Java-centric systems in some 
states, Microsoft 1 in others, aging hardware running 
COBOL in more places than preferred, and a smor- 
gasbord of commercial partners in still others. This 
fragmentation hinders efforts at connecting disparate 
data sources, not to mention moving standards-based 
curriculum and learning objects, such as test items 
and other chunks of reusable instructional content, 
between education levels and agencies. 

Same Data, 
New Uses and Users 

Many of today's state data systems are organized by 
"events" (e.g., "official" enrollment dates, state 
tests) cataloged by district and school. This snapshot 
view affords policymakers and education leaders 
cross-sectional views of how schools and districts are 
doing, that is, the aggregated performance of stu- 
dents at different points in time. Organizing the data 
by events, however, offers little actionable informa- 
tion to the leader looking to improve student 
performance, and typically there is no view of indi- 
vidual student progress. Consequently, states are 
under pressure to transition from an event-driven 
data system to one organized by individual student 
records. Such a system, which states are currently 
transitioning to, allows tracking variables over stu- 
dents' learning careers and enables analysis of 



longitudinal data for further insight. Table 2 shows 
various aspects of the transition to a more respon- 
sive data system. 

Implicit in (although not required by) the NCLB Act is 
a call for state capacity to track individual student 
achievement, a significant step for many states. Some 
states, Michigan, for example, built their statewide 
student-level databases several years ago. Indiana 
assigned unique identification numbers to all its stu- 
dents in 2003 and is building a single database in 
2004. Illinois and Wisconsin have awarded bids to 
build similar capacity. A few states cite smaller dis- 
tricts' difficulty to comply with the reporting. Others 
mention student-privacy concerns, based on the fact 
that the Family Educational Rights and Privacy Act 
gives parents certain rights with respect to their chil- 
dren's education records (U.S. Department of 
Education, n.d.). Regardless of the decision made 
about individual student records, states at this junc- 
ture are wise to consider which other related 
upgrades are desirable — and doable. 

Still, for local practitioners, standardized test scores 
cannot provide data at the depth and frequency 
necessary to inform decisions about instructional 
practice. Districts and schools need more insight 
into aligning instructional content to standards, 
strategies for teaching to individual student differ- 
ences, and guidance on resources for meeting 
goals. A tool that several SEAs are testing to link 
assessment, standards, and classroom practice is 
the Surveys of Enacted Curriculum (SEC). Porter (in 
press) describes well the possibilities for the SEC. 
Several states have made inroads into providing sys- 
tems that support a "smart desktop" level (see 
Palaich, Good, Stout, & Vickery, 2000) of decision 
making for teachers, notably the Massachusetts 
Virtual Education Space (ves.mass.edu/). Other vari- 







Data organized by events 

Static, aggregated records 

Data in discrete sets or "silos" 

Data and reports concentrated 
in hands of few 

Expertise required to use 

Numerical, canned, and columnar reports 


Data organized by individual student records 

Atomistic records, relational in time and hierarchy 

Records fully relational by field and data set 

Access to data and reporting capability at SEA, district, 
school, and classroom levels 

User-friendly access routines; continuous user training 
Graphical, interactive displays 
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ations include Web-based resource connections 
such as Ohio's Instructional Management System 
(ims. ode. state. oh. us/ODE/l MS/default. asp?bhcp=1) 
and the Curriculum Resource Center, a system built 
by Wisconsin (www2. dpi. state. wi.us/sig/practices/ 
high_2.asp). We have focused on states. However, 
district-level examples are also relevant. Chicago, 
for instance, has maintained relational student-level 
databases since the 1970s, with online school 
access to near real-time results. The Cleveland 
Municipal School District is building diagnostic 
"testlets" aligned to Ohio standards and making 
detailed, longitudinal, and real-time performance 
patterns visible to teachers. 

Finally, there exist international examples worth a 
closer look. At the University of Durham, for 
instance, a grassroots data system has grown to 
serve 5,000 British schools, providing detailed with- 
in-school, within-subject data disaggregations. 
(Fitz-Gibbon & Tymms, 2002). 

Clearly, the student-performance data picture is 
complex, but there is more. States track education 
fiscal data, student poverty statistics, English-lan- 
guage learners, technology equipment, and so on. 
The system should be viewed as part of a larger 
state data infrastructure, with data moving among 
departments, back down to districts and schools, 
potentially between states, and, of course, to feder- 
al summary databases. State data strategists, 
therefore, need to consider functionality within and 
data exchange across these contexts. To be fair, 
information technology staff in most states recog- 
nize the need for making data accessible and 
presentable to various users, but policy priorities 
and limited state budgets preclude them from 
addressing these needs more fully. Keeping the 
focus on end-user utility and affordability is difficult 
when the end user is not in the conversation. 

New Kinds of 
Performance Measures 

The NCLB Act requires states to collect and report 
on new performance measures, and some districts 
are interested in yet other kinds of data. Just as aca- 
demic standards vary from state to state, there are 
multiple views on which performance measures offer 
the best views on student learning. While the NCLB 
Act and some state accountability systems empha- 
size yearly grade or school improvement, some 



researchers argue that year-over-year test scores are 
subject to too much volatility to serve as reliable 
measures of school performance (Kane & Staiger, 
2002; Linn & Haug, 2002; Rogosa, 2002). Data sign- 
posts for state and federal policy targets may not 
align with local needs, spurring inappropriate or inef- 
fective action. For instance, almost every state now 
identifies a few former "flagship" schools as in need 
of improvement. Whether and how to reallocate 
resources in such instances is not a simple decision. 

The point is often made that local leaders need 
finer-grained data on student performance, and 
they need it more frequently. Some see the required 
NCLB statistic — percentage of students above pro- 
ficient — as insensitive to variations within buildings 
and across time. Nor does it yield insight into 
whether value is being added rapidly enough. 
Reporting, and thereby inferences drawn from the 
data, are affected by which scores are used and how 
they are analyzed (Seltzer, Frank, & Bryk, 1994; 
Wainer, Hambleton, & Meara, 1999). A stronger focus 
on patterns of individual student growth over time 
rather than proportions of students meeting standards 
is strongly advocated by a variety of researchers 
(Sanders & Horn, 1994; Seltzer, Choi, & Thum, 2003). 
The answers data provide are also strongly influenced 
by the questions asked. A more principled approach 
to thinking about teaching and learning, and learning 
about data, is needed (Herman, Baker & Linn, 2004). 

Demand for alternatives to standardized tests (e.g., 
performance and portfolio-type assessments) means 
states must develop more sophisticated strategies for 
recording and analyzing these kinds of assessment 
data, in addition to other reporting requirements. 
The bottom line: As data use increases, so will the 
demand on states to manage more data and more 
types of data. 

Sharing Data and 
Accountability 

As mentioned, data resided in discrete domains in 
the early days of electronic data gathering. Further, 
data tended to flow upstream, from schools and 
districts toward the state and federal levels. 
Compliance reporting needs drove (and continue to 
drive) much of the state investment in data systems. 
Today, educators and researchers want these data- 
bases integrated in such a way that data can be 





shared between domains and across levels of users. 
This is not yet a reality in most states. 

Districts that use data effectively for decision making 
and school improvement give frequent informal tests 
between high-stakes state testings (Armstrong & 
Anthes, 2001). This gives instructors data that can be 
used to diagnose gaps in student learning, which 
allows them to design interventions to close the 
gaps (Smith, Smith, & DeLisi, 2000). States rethinking 
and redesigning education data systems then have 
an opportunity to empower local educators and dis- 
tribute accountability throughout the system 
(Cromey & Hanson, 2000; Massell, 2001). Once 
states have data systems with the appropriate level 
of detail to aid in improving instruction, they can 
begin to create a system of support for schools 
struggling to meet adequate yearly progress goals, 
using scientifically based research data. 

Interstate efforts can also facilitate the flow of data 
for comparison and decision making. Just for the 
Kids, a Texas-based nonprofit organization, uses 
data provided by SEAs from 17 participating states 
to identify how well schools are doing and to provide 
information on the best practices of high-performing 
schools. In 2003, Standard & Poor's School 
Evaluation Services (SES) provided Michigan and 
Pennsylvania with school-level data disaggregation, 
with particular focus on student performance and 
operating cost, calculating a variety of indices to 
estimate educational productivity. Recently, SES and 
Just for the Kids jointly received funding from the 
Broad Foundation and the U.S. Department of 
Education to provide disaggregated student 
achievement data and data analysis tools to states. 
This public-private collaboration, the School 
Information Partnership, is focused on the data 
being reported for NCLB requirements, and its 
online tools (at www.schoolresults.org) currently pro- 
vide analyses for a growing number of states. 

The growing market for data tools and support serv- 
ices for data use attests to the desire of schools and 
educators to use data effectively for improvement. 
For example, six states, including Illinois, Indiana, 
and Ohio, participate in the Baldrige in Education 
Initiative, aimed at giving educators a framework and 
tools for implementing continuous improvement 
strategies. In Ohio, Battelle for Kids provides a similar 
focus. Connecting these strategies and data into state 
systems will be a challenge, and a necessity. It is 
unlikely that any SEA system can fully provide for the 



data needs of all participants in the education enter- 
prise — and even the most ambitious SEA technical 
staff should accept this. 

Toward Data 
Standards 

The trends toward open computing and common 
protocols made Internet communication possible 
and made business computing much easier and 
more efficient. A parallel effort is currently underway 
to make education data sharing easier. Partners in 
the quest for data-sharing solutions span industry, 
government agencies, and education organizations. 
For example, Schools Interoperability Framework 
(SIF, 2004) and Common Object Request Broker 
Architecture (CORBA) (Object Management Group, 
2004) establish conventions that smooth the way for 
the task of liberating data from legacy or foreign sys- 
tems. Several organizations link information on state 
standards, helping to bridge stand-alone data sets. 

From a national perspective, state solutions to data 
management have tended toward the iconoclastic, 
leaving little room for economies of scale or sharing 
good applications. In 2002, the U.S. Department of 
Education funded the United States Open e- 
Learning Consortium (USOeC), a 14-state 
consortium to explore the use of statewide, interop- 
erable e-learning and decision-support platforms 
(Center for Teaching, Learning, and Technology, 
2001-2002). If the USOeC reaches its goals, states 
could more easily share data, learning objects (e.g., 
test items and lesson plans), and tools to support 
standards-based, curriculum-aligned education. 
However, creating such an open platform takes time 
and interstate cooperation, two commodities that 
are not in abundant supply. 

Another U.S. Department of Education program is the 
Performance Based Data Management Initiative 
(PBDMI). (Although this program is an initiative of the 
department's Office of the Chief Information Officer, 
the most detailed information about this effort resides 
on the Web site [evalsoft07.evalsoft.com/pbdmi] of 
principal contractor Evaluation Software, Inc.) PBDMI 
seeks to set standards, guidelines, and policy on data 
collection and information management practices, 
while moving the focus of data collection away from 
compliance and toward performance. The work is 
expected to result in the creation of the Education 
Data Exchange Network (EDEN). 
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Learning Point Associates offers Data 
Retreats to schools and districts nationally. 
These retreats provide a comprehensive 
professional development framework that 
(1) addresses collection and analysis of 
data; (2) enables educators to analyze the 
data, using a generic and supportive 
process to identify and prioritize goals 
and strategies; and (3) allows evaluation of 
solutions to sustain and build a data culture. 

The National Center for Education Statistics (NCES) 
has supported the National Forum on Education 
Statistics for over a decade. The forum brings togeth- 
er data-system specialists from all 50 state agencies 
once or twice each year. These meetings have pro- 
duced a valuable series of working papers and 
standards documents (see www.nces.ed.gov/forum). A 
core piece has been a set of data handbooks and sup- 
port tools that focus on the needs of school-, district-, 
and state-level administrators. Examples of these pub- 
lications include the Student Data Handbook for 
Elementary, Secondary, and Early Childhood 
Education: 2001 Update (NCES, 2001); Building an 
Automated Student Record System (NCES, 2000); 
and Protecting the Privacy of Student Records 
(Cheung, Clements, & Pechman, 1997). In 2002, the 
forum piloted an exchange of standards-based, cur- 
riculum-aligned assessment objects for use in 
formative assessments. It continues to develop a clas- 
sification system for sharing and retrieving learning 
objects based on each state's curriculum standards. 
Since 2000, the Council of Chief State School Officers 
has been convening various states to develop and 
improve state data systems. About 15 states partici- 
pate in this Accountability Systems and Reporting 
(ASR) project. The ASR project provides guidance 
around data systems and has published criteria for 
high-quality state reports, examples of state plans for 
adequate yearly progress (AYP), and strategies to 
address critical issues related to AYP. 

Clearly, progress is being made in two areas: 
creating standards for data exchange and interoper- 
ability, and building the will to adhere to these 
standards. Sharing of data and data findings should 
also be part of the thinking about data systems. 
States and districts need not all repeat the same 
experiments; more efficient replications are possible, 



and common learning will accelerate progress. While 
copyrights and patents assure intellectual rights and 
profit, scientists continue to construct methods to 
share research and research data efficiently. As states 
construct new and better data systems, they would 
be wise to study the approaches researchers have 
developed (Arzberger et al., 2004). 

IMPLEMENTATION CHALLENGES 
FOR GOVERNMENT 

The previous section examined the multifold and 
ongoing factors that strain state data-system capaci- 
ty. However, good command of data promises big 
payoffs in student achievement and school success. 
At least that is what the current clarion call for data- 
driven decision making implies (Massell, 2001 ; Serim, 
2003; Streifer, 2002). While the call sounds sensible, 
strong evidence for it in education is difficult to come 
by. Case studies abound, but only recently has credi- 
ble research begun to make the point. 

Martin Carnoy and Susanna Loeb (2002), researchers 
at Stanford University, studied states' accountability 
policies and changes in state average National 
Assessment of Educational Progress (NAEP) scores 
and found that states with stronger accountability 
systems showed greater improvement in student 
performance. Harvard's Brian Jacob (2004) conduct- 
ed a finer-grain analysis, linking changes in the 
accountability structures of the Chicago Public 
Schools to student performance and teacher behav- 
ior. Improved math and reading scores followed 
heightened accountability. 

Chrispeels, Brown, and Castillo (2000) have docu- 
mented that data use predicts more effective school 
improvement teams. Stronger accountability and 
better data engender improved performance. But 
the path that links data and performance remains 
murky: The decisions local actors will make are not 
readily predictable, and sometimes those decisions 
contravene good educational practice as short-term 
tactics "out-elbow" long-term strategy. 

This situation demands thoughtful action combined 
with expert guidance. The complexity of technology- 
based innovation, coupled with the political realities 
of adopting and implementing new programs, is 
daunting for even the most well-intentioned reform- 
ers. Very few legislators or elected officials feel they 
have sufficient knowledge about technology to 



make important investment decisions. In fact, policy- 
makers report finding little research about 
technology that is relevant to their policy questions 
and deliberations (Education Commission of the 
States, 1998). Furthermore, political bodies are not 
designed to sort out the technical details of an inte- 
grated and robust state education data system. 

On the other hand, state agency technical staff and 
managers lack the time, sometimes the vision, and 
most often the political know-how to guide the 
oversight and governance of technology invest- 
ments. Despite the broader vision of chief 
information officers (CIOs) — for those state agencies 
that have appointed them — even CIOs may not be 
well equipped to deal with multiple legislators on 
thorny technical issues. And, most CIOs are not 
trained in the complexities of education data analy- 
sis and data utilization, because most mastered their 
craft in business environments. These professionals 
need to be at the decision-making table, but it is not 
clear that they should lead the process. 

For other complicated and technical issues such as 
utilities, transportation, and energy, legislators vote 
on policy based on the recommendations of com- 
missions or boards appointed for their expertise and 
varied perspectives. These boards listen to the con- 
cerns and advice of stakeholders from across the 
state and political spectrum, as well as the needs 
and concerns of consumers. They consider multiple 
options, work out compromises, and present legisla- 
tors with plans on which to vote. 

States need a similar body to recommend action 
across the full spectrum of a state's education data 
system. Rather than requiring a task force, data sys- 
tems require an ongoing commission, appointed by 
the legislature and the governor, with SEA represen- 
tation, and with a clear agenda. The commission 
should oversee five key data tasks that states must 
perform. Thorn (2001) identifies three of those tasks: 
define and prioritize what to study and/or measure, 
manage the data (infrastructure, security, access), 
and measure performance meaningfully. States must 
also report the data in useful and timely ways and 
build the technical and human capacity to use the 
data effectively. 

The commission would conduct conversations with 
the education agency, the legislature, and other 
agencies dealing with children to ensure that the five 
aforementioned tasks are meeting stakeholder 
needs. The commission should have the capacity to 



Despite the broader vision of chief 
information officers (CIOs). ..most CIOs 
are not trained in the complexities of 
education data analysis and data 
utilization, because most mastered 
their craft in business environments. 

make decisions, develop a comprehensive plan, and 
present recommendations to the legislature for vote. 
The scope of the commission's work should encom- 
pass K-12 public education. Each state should 
decide for itself whether to include representatives 
at other levels of education (preschool, higher edu- 
cation, nonpublic education) on the commission. 

POLICY RECOMMENDATIONS 

State by state, SEAs have been moving forward 
along the road to data sophistication — if in fits and 
starts. Given states' fiscal uncertainties, it is tempt- 
ing to delay a comprehensive examination of data 
infrastructure and management until state coffers are 
full. Mistakes are costly, so too is inaction. 
Incremental measures can extend the lifespan of an 
existing system, but states run the risk of spending 
more to get less in the long run. It is more prudent 
to develop a coherent data systems plan and a 
mechanism for overseeing its implementation. 

The following recommendations summarize this 
report's findings and provide options for policymak- 
ers to consider. 

• Address the key tasks associated with effective 
use of education data. There are five such 
responsibilities: 

- Define and prioritize what to study and measure. 

- Ensure that student, school, and system per- 
formance are measured meaningfully. 

- Manage and integrate disparate data sources 
(infrastructure, security, access). 

- Report the data in useful and timely ways. 

- Build the technical and human capacity to use 
the data effectively in the schools and centrally. 

• Create an education data oversight commission 
in each state. These permanent commissions 
would manage each state's education data system — 
regulating and overseeing its development — and 
recommend policy, planning, and investment deci- 
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sions to the legislature. The individual state com- 
missions should themselves have opportunity to 
coordinate with each other and with federal data 
leadership. 

• Develop common data standards, data-collection 
guidelines, and data-management practices. 

Representatives from the SEA or the newly creat- 
ed commission should participate in this 
multistate effort. 



CONCLUSION 

Ever since states committed to standards-based 
reform, they have been confronting data-capacity 
issues. States manage data about standards imple- 
mentation (accountability measures); they manage 
test-score data on how much students are learning 
(performance measures); and they manage data 
about the test takers and their schools (context and 
demographic data). Outside pressures also point to 
data mastery. ESEA encouraged data management 
practices in the name of accountability; NCLB legisla- 
tion in effect requires it. 



building a robust data system is essential to making 
progress on education reform. In each state, a coali- 
tion, led by the governor, the state superintendent 
or key legislators, or some combination thereof, 
must initiate the effort. The coalition must appoint a 
commission with a clear agenda and the authority to 
ensure that states accomplish key data tasks. 

Efforts to establish data standards and conventions 
will facilitate data use for decision making and will 
reduce inefficiencies. Data, and the costs of manag- 
ing it, can be shared. Several such efforts already 
exist (e.g., SIF in the private sector, PBDMI and the 
NCES data forum in the public sector). However, 
their sense of urgency needs to grow, and other 
voices need to join. Strong support from political 
actors will help force the agendas. 

These efforts should also be coordinated with the 
work of the What Works Clearninghouse and other 
efforts to make educational knowledge more rigor- 
ous and more evidence based. The administrative 
and performance data sytems, standards, and con- 
ventions that state systems design and implement 
should in the long term also simplify the collection of 
data for research purposes. 



States need to adopt a proactive, comprehensive 
approach to building and maintaining the education 
data system. Key policymakers need to agree that 
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